[PIO-209] Upgrade Elasticsearch to 6.8 for pre-built binary distribution #516

shimamoto · 2019-06-13T10:14:47Z

Most users use pre-built binary distribution and would like to run on new versions as much as possible.

before:
Spark 2.1.3
Elasticsearch 5.6.9
Hbase 1.2.6

after:
~~Spark 2.4.3~~ ---> separate a PR
Elasticsearch 6.8.1
~~Hbase 1.4.10~~ ---> separate a PR

The dependencies are complex, so I will try to handle tasks separately.

This PR upgrade Elasticsearch to 6.8 and deprecate Elasticsearch 5 because of EOL (https://www.elastic.co/support/eol).

shimamoto · 2019-07-04T11:27:04Z

Currently, in the develop branch (Elasticsearch 6.4.2), there are no HEAD request warnings.

[INFO] basic_app_usecases 2019-06-04 02:28:26,772: Adding a new application
[WARN] [RestClient] request [PUT http://elasticsearch:9200/pio_meta_apps] returned 1 warnings: [299 Elasticsearch-6.4.2-04711c2 "the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template" "Tue, 04 Jun 2019 02:28:29 GMT"]
[WARN] [RestClient] request [PUT http://elasticsearch:9200/pio_event_1] returned 1 warnings: [299 Elasticsearch-6.4.2-04711c2 "the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template" "Tue, 04 Jun 2019 02:28:30 GMT"]
[INFO] [App$] Initialized Event Store for this app ID: 1.
[INFO] [Pio$] Created a new app:
[INFO] [Pio$]       Name: MyRecommender
[INFO] [Pio$]         ID: 1
[INFO] [Pio$] Access Key: xPsroclOrmAViB0OA7Q8EsYX_uRNOPvPLLGMEL6coSmN8hFA-F4ouGwETCxQvF1p

When upgrading Elasticsearch to 6.8.1, I have run into the following warnings:

pio@b507ffc58515:/PredictionIO-0.15.0-SNAPSHOT/MyRecommendation$ $PIO_HOME/bin/pio app new MyApp1
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_sequences] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_sequences/_mapping/sequences] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_apps] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_apps/_mapping/apps] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_event_1] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [PUT http://localhost:9200/pio_event_1] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template"]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_event_1/_mapping/1] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [PUT http://localhost:9200/pio_event_1/_mapping/1] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] Specifying types in put mapping requests is deprecated. To be compatible with 7.0, the mapping definition should not be nested under the type name, and the parameter include_type_name must be provided and set to false."]
[INFO] [App$] Initialized Event Store for this app ID: 1.
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_accesskeys] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_accesskeys/_mapping/accesskeys] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[INFO] [Pio$] Created a new app:
[INFO] [Pio$]       Name: MyApp1
[INFO] [Pio$]         ID: 1
[INFO] [Pio$] Access Key: pYqtHCiLzYwTyiodCAFC_cekeAfy-k8l0Fta4yWWvC42-dEB9piso5rPnzXCsP90

pio@b507ffc58515:/PredictionIO-0.15.0-SNAPSHOT/MyRecommendation$ $PIO_HOME/bin/pio app list  
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_sequences] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_sequences/_mapping/sequences] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_apps] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_apps/_mapping/apps] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_accesskeys] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[WARN] [RestClient] request [HEAD http://localhost:9200/pio_meta_accesskeys/_mapping/accesskeys] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in get mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
[INFO] [Pio$]                 Name |   ID |                                                       Access Key | Allowed Event(s)
[INFO] [Pio$]               MyApp1 |    1 | pYqtHCiLzYwTyiodCAFC_cekeAfy-k8l0Fta4yWWvC42-dEB9piso5rPnzXCsP90 | (all)
[INFO] [Pio$] Finished listing 1 app(s).
pio@b507ffc58515:/PredictionIO-0.15.0-SNAPSHOT/MyRecommendation$

That's why tests fail. When executing pio commands, the HEAD request warnings will always be visible to users.

@marevol @dszeto How should we deal with that?

Wei-1

Elasticsearch 6.x

In 6.8, Not setting include_type_name will result in a deprecation warning.

Elasticsearch 7.x

The include_type_name parameter in the index creation, index template, and mapping APIs will default to false. Setting the parameter at all will result in a deprecation warning.

Elasticsearch 8.x

The include_type_name parameter is removed.

Given the context on the Elastic Search Website
I think it makes sense that we bypass this warning.

I believe we should modify the show function in utils
Since using the number of lines to determine whether it is an id, name, or description might be too risky, as we might face similar issues in the future.

shimamoto · 2019-07-05T03:08:32Z

@Wei-1 Thanks! This is good info.

I would like to hear other committer's judgement on this, and work out details.

shimamoto · 2019-07-05T03:26:33Z

The Elasticsearch team has been extremely aggressive with breaking changes continuously.

shimamoto · 2019-07-17T02:48:28Z

We have to remove 5.6 support before supporting 7.0 because method performRequest(String method, String endpoint) is removed as of ES 7.0.

shimamoto · 2019-07-17T02:51:12Z

@dszeto @marevol @takezoe please review and let me know for any suggestions. Thanks

Unless I hear from you to the contrary, I will merge at the end of the month.

takezoe

It looks very good to me. Great work!

morrismatts · 2019-10-30T03:34:05Z

Does this PR need anything more? Or is it ready to merge?

Thanks! 👏👏👏

EmergentOrder

LGTM

shimamoto added 8 commits June 13, 2019 19:10

Upgrade Spark to 2.4 for pre-built binary distribution

d9a1b58

fixup

3aff7a5

fixup

8d4c68b

fixup

9a6e709

fixup

7f538ce

fixup

07b56de

Revert hbase

94b77e5

fixup

8310866

shimamoto changed the title ~~[PIO-209] Upgrade Spark to 2.4 for pre-built binary distribution~~ [PIO-209] Upgrade Elasticsearch to 6.8 for pre-built binary distribution Jun 27, 2019

shimamoto added 3 commits July 4, 2019 11:55

fixup

cadd372

fixup

c044311

fixup

f9060af

shimamoto requested review from marevol, dszeto and takezoe July 4, 2019 11:29

Wei-1 reviewed Jul 5, 2019

View reviewed changes

shimamoto added 3 commits July 16, 2019 18:42

fixup

2a61b76

fixup

d9b2f65

fixup

4bd1d9a

shimamoto mentioned this pull request Jul 17, 2019

[WIP][PIO-209] Upgrade Spark to 2.4 for pre-built binary distribution #521

Open

takezoe approved these changes Jul 17, 2019

View reviewed changes

EmergentOrder self-requested a review November 5, 2019 15:19

EmergentOrder approved these changes Nov 5, 2019

View reviewed changes

EmergentOrder merged commit 9ed7d84 into apache:develop Nov 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PIO-209] Upgrade Elasticsearch to 6.8 for pre-built binary distribution #516

[PIO-209] Upgrade Elasticsearch to 6.8 for pre-built binary distribution #516

shimamoto commented Jun 13, 2019 •

edited

Loading

shimamoto commented Jul 4, 2019

Wei-1 left a comment •

edited

Loading

shimamoto commented Jul 5, 2019

shimamoto commented Jul 5, 2019

shimamoto commented Jul 17, 2019

shimamoto commented Jul 17, 2019

takezoe left a comment

morrismatts commented Oct 30, 2019

EmergentOrder left a comment

[PIO-209] Upgrade Elasticsearch to 6.8 for pre-built binary distribution #516

[PIO-209] Upgrade Elasticsearch to 6.8 for pre-built binary distribution #516

Conversation

shimamoto commented Jun 13, 2019 • edited Loading

shimamoto commented Jul 4, 2019

Wei-1 left a comment • edited Loading

Choose a reason for hiding this comment

Elasticsearch 6.x

Elasticsearch 7.x

Elasticsearch 8.x

shimamoto commented Jul 5, 2019

shimamoto commented Jul 5, 2019

shimamoto commented Jul 17, 2019

shimamoto commented Jul 17, 2019

takezoe left a comment

Choose a reason for hiding this comment

morrismatts commented Oct 30, 2019

EmergentOrder left a comment

Choose a reason for hiding this comment

shimamoto commented Jun 13, 2019 •

edited

Loading

Wei-1 left a comment •

edited

Loading